[Feat] Make the decision deadline configurable for slower local System One servers - #36
chaimaerachdi wants to merge 3 commits into
Conversation
JevModel cut every decision at 5 s, a deadline sized for Jev (0.3 s). A local System One server behind TYPESAFE_API_URL can be slower: on Google Flights, OneJev-27B on an A100 takes about 3.7 s a decision and more on the calendar page, so every run stopped at the eighth step. With 30 s it completed the task with the same 11 actions as Jev. S1A_DECISION_TIMEOUT_S sets the deadline; unset, it stays 5 s, so Jev is unchanged. A value that is not a positive number is a config error. An explicit timeout_s passed to JevModel.from_env still wins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| value = float(raw) | ||
| except ValueError: | ||
| value = 0.0 | ||
| if value <= 0: |
There was a problem hiding this comment.
float() accepts nan, inf, and overflow such as 1e9999; neither nan nor positive infinity satisfies value <= 0, so they pass this guard. nan then makes the computed deadline and remaining timeout nan, while inf removes the intended bound. Please also reject not math.isfinite(value) and extend the config-error regression with nan, inf, and 1e9999.
float() reads nan, inf and 1e9999; nan passed the `<= 0` guard and made the deadline nan, inf removed the bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Non-blocking:
FYI, out of scope for this PR: an explicit |
|
@chaimaerachdi could you add a Demo / evidence section using the self-review guidance now merged in #39? The measurements in the description are helpful; a short recording, screenshots with logs, or a reproducible terminal trace would make the deadline change easier to verify. On the same task and local System One server, please show:
Please include the invocation/config, model and server versions, hardware, exact baseline/head commits, actual elapsed times and outcomes, and links to the raw traces. For the reported Flights comparison, retain the three interrupted runs in the run record alongside the seven completed runs. Label any injected-delay fixture separately from real-server evidence. Existing artifacts are fine; no new A100 run or video is required. If a real-server run is unavailable, note the gap. Please redact secrets/private data and say whether any attached media may be reused in public project updates, with any attribution requirements. |
…_TIMEOUT_S Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the review. Updated docs/browser-front.md in de5f5d2 to mention S1A_DECISION_TIMEOUT_S. The explicit timeout_s validation can go in a separate PR, as suggested. |
4 participants
Why
JevModelcuts every decision at 5 s (DECISIONS_TIMEOUT_S), a deadline sized for Jev, which answers in about0.3 s. A local System One server reached through
TYPESAFE_API_URLcan be slower. On Google Flights, OneJev-27B(OmniJev's new 27B model, which speaks the same API) on an A100 takes about 3.7 s a decision and more on the calendar
page, which offers 66 more elements. Every run stopped at the eighth step, right after the calendar opened.
How
s1a/decision_models/wire.py:decisions_timeout_from_env()readsS1A_DECISION_TIMEOUT_S; unset, it returnsDECISIONS_TIMEOUT_S(5 s). A value that is not a finite, positive number (nan,infand1e9999included) isMODEL_SERVICE_CONFIG_ERROR, naming the variable.s1a/decision_models/jev.py:JevModel.from_env(timeout_s=None)uses it; an explicittimeout_sstill wins.What
S1A_DECISION_TIMEOUT_S(docs/configuration.md,.env.example,CHANGELOG.md).Measured (Google Flights, Zurich to London, one way; A100 80 GB; headless Chromium)
Success is judged by a vision model (Gemini 2.5 Flash) on the final screenshot. Time is split with the repo's
profiler (
--profile-out), as indocs/benchmarks.md.qev serve)qev serve)Times are medians over the runs. At 30 s, ten runs were started and three were interrupted by hand before the end;
those three are not counted.
Verification
uv run ruff format --check . && uv run ruff check .: clean.uv run ty check: 1 diagnostic ins1a/decision_models/cua.py, the same onmain.uv run pytest -q --ignore=tests/test_browser_policy.py: 394 passed, 16 failed;mainon this Windows machinefails the same 16 (environment), so none is new. 4 new tests in
tests/test_decision_models_jev.py(
TestDecisionTimeout): default 5 s, the variable, an explicit value winning, invalid values (abc,0,-3,nan,inf,-inf,1e9999).